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‘Does evaluation really shape social ‘tity that's the aucune: or are we 


walking the fine line of political’ decision making with or without data? 


= 


Cronbach (1980) points eon that our political and organizational 
realities determine what evaluation can do. ‘Mbeonderataadiagas of the 
pallies making process are a source of not pay faulty evaluation 
practices and criticism of sound practices, but also are a source of 


/ . 
faplty data use practiced. Recommendations for evaluation and data use 


should derive from a sound understanding ofhow and why actions are 


taken. podietoually, administrative and isa connections among the 
key actors as well as the political ‘forces to which the actors are 
sensitive strongly influence the course of evaluation and the use of he 
insuing information. It should be pointed out that strictly honest data 
collections can also generate misleading pictures unless questions are 


framed to expose the facts which are useful to program supporters and 


detractors. “ 


cba haia and research data have been used/misused repeatedly in the 
determination and shaping of adveationat-ailicy and in the resolution of 
educational funding and political decisions. In debates over 
controversial programs “liars figure end figQres often lie.” The 
evaluator and the shaper of social policy have a responsibility to 
protect nia/her clients from both types of deception. One might 
efgk¢fully ask what is the interaction of policy and decision making? 


What is a policy question? Green (1982) points out that neither the most 


efficient action nor the most technically proficient analysis are precise 


enough to resolve the central conflict between the social aims which give 


rise to a policy question. ‘ « Ae 


Wise policy is not made with enough knowledge aeega a decision, 


and policy qugattong, are never iskea out of a prima interest in adding 


\ 


to our knowledge. It can be argued that our answers to policy questions 


may be“ improved by obtaining better information and by doing SPREBEE 


analyses which will then be more rationally vatauaedve: That aia 


. a 
questions, can, will and usually are answered even without such 


information goes without saying. Evaluation findings can never 


completely determine a position. 

P ‘ | | . 
One basically needs to look at the policy process in terns of, the entire 
decision ‘making framework. The policy process is comprised of the policy 
analysis stage, the policy formation stage, the policy decision stage, 
and the political analysis stage (Green, 1982). Policy: anglysis can be 
defined as he rational or technical assessment of the net marginal 
tradeoff between different policy choices. The question becomes 
(a) which set of values will be advanced, (b) which will not, and 
Cc) mate, rd net. benefits? Setting forth the marginal costs and 
benefits of a range of choices is one thing -- policy# analgata, 
Selecting one balanced choice or 4 eats from within the possibilities is 
another -- policy fisdaeione The decision as to'which choice or choices 
will be made is still a third -—- policy decision, and performing the 
market analysis needed for that decision is political analysis. 

In short, poldcy sudivei is : thag rational, technical analytic 
performance in which the central question jis ok whether X is a good, 


thing to do but simply what are the marginal effects of doing X and what 


are the marginal effects as contrasted with doing something else. 
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‘Evaluators can take a iecdtus role: in this iii provided ‘that they do 
not suppose they are onal evaluating policy as opposed to udrely | 
recording the consequences of doing X or Y. ‘Policy formation on the 
other hand is that activity -by which we seek to sain Keceudent on what 
form a specific policy can or will tke as dapomed o what forn fe avai 
to take. Sy contrast, policy Aesinies ean’ be described as the 
authoritative action of some office, administrative or legislative,. by 
which a line of action for the moment at least is established. Policy 


dncigten is not so much an etiviey or process as it is a thomentary pornt 


in the continuing business of government. 


Political analysis is concerned not with: determining the net benefits of 
a given course of action, but with their political weight. The aim is 

not so much to determine the net social benefits of a particular policy, 
>but to determine its constituency. The rational standards of policy 
analysis are the akaiiatts of theoretical reason, but the rational 
standards of policy decision and political analysis are the standards of 
political judnewsants In short, the exercise of peiteiaal judgement is a 
practical activity. It is also an“@valuational activity, but the result 
of that activity may differ from se even contradict the results of policy 
analysis. For researchers and evaluators, political and legal 
considerations are just as important as_ technical considerations 
depending on the level of evaluation. For example, at the state 
educational level, evaluation wibeneien and consideration is Peuaalnts 
focused on management assistance Ge policy snaiyete to the general 


exclusion of the improvement of instruction, Because of such 


orientations anf considerations, the ability to communicate and persuade 


: as 


in a highly politicized] environment becomes an essential skill. In 


addition, hudgetary d financial analysis, problem definition, 


understanding of the context and the ability to know what can be effected 
and how within the organizational setting are also needed for effective 
evaluation within state| agencies or whatever contextual environment. 
Green (1982) suggests that “If we just had’ a methodology sufficiently 
sophisticated and a body \of relevant data sufficiently refined we could 
ee whatever policy questions may, come along.” If. this were true; 
such a person obviously hag been captured by a delusion. The delusion is 
in supposing that our polisy. question is a RMEDER EASE question when in 
fact it is not. Any tine we suppose chat a soley question can be 
Yesolved by some addition to our knowledge, then it will turn out that 
“what we supposed was a question of policy G a theoretical one and has 


‘turned out to be merely a problem of engineering or efficient 
‘i 4 


administration instead. We should not however abandon all attempts to 
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improve methods of evaluation in policy analysis. Since our indecisioneg 


in matters of policy does ‘not arise from the lack of such methods, 
therefore, it is unlikely to be laid to rest by their development. L 
policy we are confronted with indecision not because our knowledge a 
technical facility is faulty but precisely because we are confronted me 
a kind of question that in principle cannot be answered with any 
increment or improvement of knowledge. Our answers to policy questions 
may be improved by better information and better analysis in the sense 


that they will be more rationally persuasive, but such questions can, 


will and usually are answered even without such information. 
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Insofar ‘that ‘the evaluator presses for greater certainty and setiatiy 
jiten to become the determiner of policy he/she is less of a professional 
at his/her best than the politician or executive at his/her best. The 
drive of evaluators to whatever extent they seek to find. in wilgcliea a 
means of resolving policy questions is in fact a drive to make the 
politician ae evaluator which is to say a technician of netiey 
decisions. Such an achievement, if aver pwnd leeds would constitute tha 
most radical transformation of our political institution and the practice 
of policy decision that one can Geeta: It should be stated that the 


evaluator who expects objectivity suffers his/her greatest “ghock 4h 


discovering that his/her information is reshaped for political purposes. 


s 
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It is interesting to note and recognize how unlikely it is that political 
contenders will ever be silenced by a factual evaluation report. We must — 
also conclude and see how unlikely it is that an evaluative conclusfon 
will unite all pdrties behind the same option. It might be interesting 
to point out the headstart evaluation as a well known éxample of (a) 
selective belief of information and (bd) attack and counter aitack: 


Williams (1969) saw its reception as typical of the kind received by an 


impact appraisal that arouses political interest. 


"The history of headstart evaluation is a stark illustration of 


. . 


what might be termed the implication of the iron law of the 


absolute evaluation ’ flaws. That is, as a general rule, the 


absolute methodological and logistical decisions in any 


evaluation can make political infighting near certainty when 
! “ 


evaluation results threaten a popular program.” 
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In short “questionable valuation practices” can always be attacked 

on methodological grounds for political and bureaucratic purposes. 

For the analyst, these deci§ions mean that even a relatively sound 

evaluation can easily cause*® severe controversy and the battle, 

although express d in thé jargon of the professional statistician, 
\ 


can be nasty. For example, with hints of foul play, or at least the 


— 
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claim that whe evaluator is insensitive to program needs and 
problems, the westtaghouss example serves a good ese te point on 
the ferocity of the struggle. Tense situations will arise that 
require great sensitivity, good timing, good Judeaeue and probably a 
good amount of pure luck in order to survive the political battle. . 
It is not: certain that the analyst will acquire through see Coeidal 
training dr<eepertence ‘the “att” of bureaucratic infighting required 
to be effective. If the ves law of absolute evaluation flaws 
holds, one may well ask from the perspective of mee interested 
parties whether studies such as the Westinghouse’ evaluation 


primarily summative was and more importantly is really worth the 


effort. 


Basically, let me say that only the “best of techniques", random 
assignments, strict sampling, objective measurement, F rigorous 
fugereuee? or aceuee, will not provide magic armor for the 
evaluation. A researcher and evaluator can only make his/her 
conclusions invulnerable by limiting them strictly to what the 
research operation showed and by attaching an index of statistical 


uncertainty. Only with this armor will you keep an evaluation study 


out of the realm of the political universe. One should remember 
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that the audience of an evaluator’s report are officials whdse 
perspectives are continually changing ~~ not only whose’ perspectives 
are continually changing but whose constituency might be continually 
changing. Clear choices among alternative programs really ave 
_ seldom made on the basis of evaluation. Rather, it's a kind of 
course of least resistence that is followed. Sad, Bue true, for the 


& 


professional evaluator. 


The evaluator sometimes may also wish to be ha Strictly 
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honest data collection can generate misleading pictures when 


qdeecione are not framed with the intent of exposing the facts which 
are both useful to partisans of the program and those useful to 
critics. An example might be the time honored practice of choosing 
items for educational assessments that about half of the students 
are expected to pass. ‘Sueh items are more powerful for ranking 
students than are easy or hard items. NAEP adopted, another policy 
apectfically for the sake of "even wMandiness". Its tests at a given 
age deliberately included items that about 90% of the students were 
expected to pass in order to give warranted port to educators who 
want to advertise school success. A only a few students 


4 . ‘ 
were expected to pass were included also as a way to disclose 


shortcomings (Finley and Berdie, 1970). 


Clearly, evaluation in terms of a political ae is universal © 


for it involves the use of information in changing power 


relationships. Additidmally, Carol Weiss (1972) “points out and 


reiterates the same saying that “evaluation has always had explicity 
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political overtones, it is deeieaee ee yield conclusipns about the 
worth of programs and in’ aia so is intended to effect the 
allocation of resources”. This. function of evaluation, as 

handmaiden to policy, is probably “the characteristic of evaluation 


research that has attracted competent researchers despite all the 


discontents and disabilities of practice. 


Additionally a great number of researchers feel that ‘fopaal 
evaluation is an inherently political process and it has in sek 
instances even greater policy consequences than do bond etecttniia: 
seeitieait decisions regarding evaluation, that is (a) shine to 
evaluate (b) how, (c) when and (d) by whom, are made on the basis of 
the .political values and resources of those including che évaluatovs 
themselves involved in any given system. Evaluation practitioners 
also concur with the body of knowledge about the political uses of 
evaluation. Evaluation obviously soccurs in a political environment 
and evaluators are participants in the seoeee ei for stakes often 
influencing positions but never wholly determining decisions. 
_Basically, evaluators should becoue influencers .on the political 
scene. If evaluations, are to be used in decision making and program 
improvement, ‘the sg eTaatioe has to become more than a mechanic or 
tinkerer playing with numbers which are proxies and often poor 
proxies for outcomes of programs. To evaluate a program 
effectively, the evaluator needs to know the nature of the program, 


the reason the progtam was established and the motivations of ‘the 


political entities and policy makers. ' Evaluators can assist in 


collaboratively developing appropriate questions to be investigated 
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based on a good understanding of the objectives of the program. 
This may. not be such an easy task since many ee objectives are 
not explicit in the policy statement. There are usually several 
political motives at play and questions are ‘not necessarily common 
to all the players in the game. Determining what to evaluate can be 


very difficult. | % 


As evaluation technology moves into the 1980's, evaluators and 
researchers need to’ bridge the gap and bring data and decision 
making into harmony, i.e. evaluation will become more political and 
politics will become more evaluative. The evaluator will eecas 
more of a politician for social policy we the politician more of an 


evaluator. 
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